Durable Emitter - TTL budget metric#2239
Conversation
There was a problem hiding this comment.
Pull request overview
This PR refactors DurableEmitter queue-expiry pressure observability by replacing the “near expiry count” signal with a more direct “TTL budget” signal (time remaining before the oldest pending event reaches EventTTL). This flows through the observer interface, store implementations, and metric instruments.
Changes:
- Replaced
NearTTLCount/NearExpiryLead-based observation withTTLBudget(EventTTL - OldestPendingAge) inDurableQueueStatsandObserveDurableQueue. - Updated Postgres + in-memory observer implementations and the emitter metrics loop to use the new observer signature.
- Renamed/retyped the gauge from
durable_emitter.queue.near_ttl(int) todurable_emitter.queue.ttl_budget_seconds(float).
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pkg/durableemitter/store.go | Computes TTLBudget from the oldest pending event instead of querying a near-expiry count. |
| pkg/durableemitter/observable_store.go | Updates DurableQueueStats and DurableQueueObserver interface to use TTLBudget and removes nearExpiryLead. |
| pkg/durableemitter/durable_emitter.go | Removes near-expiry lead plumbing and updates metrics loop to call the new observer signature. |
| pkg/durableemitter/durable_emitter_test.go | Updates test in-memory store observer to compute TTLBudget. |
| pkg/durableemitter/durable_emitter_metrics.go | Introduces the durable_emitter.queue.ttl_budget_seconds gauge and records it from observed stats. |
Comments suppressed due to low confidence (1)
pkg/durableemitter/durable_emitter.go:788
- When PollInterval is unset/zero, metricsLoop currently defaults to polling every 500ms, but DurableEmitterMetricsConfig.PollInterval is documented as "Zero = 10s". This mismatch is likely to surprise callers and can significantly increase DB load due to ObserveDurableQueue being called 2x/sec by default. Consider aligning the implementation with the documented 10s default (or update the doc if 500ms is intended).
mc := d.cfg.Metrics
poll := mc.PollInterval
if poll <= 0 {
poll = 500 * time.Millisecond
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if m.queueTTLBudgetSec, err = meter.Float64Gauge( | ||
| "durable_emitter.queue.ttl_budget_seconds", | ||
| metric.WithUnit("s"), | ||
| metric.WithDescription("Seconds of TTL headroom for the oldest pending event (EventTTL - oldest age); low/negative → DLQ/expiry pressure. Alert engine decides what 'near' means"), | ||
| ); err != nil { |
| queuePayloadBytes metric.Int64Gauge | ||
| queueOldestAgeSec metric.Float64Gauge | ||
| queueNearTTL metric.Int64Gauge | ||
| queueTTLBudgetSec metric.Float64Gauge |
There was a problem hiding this comment.
Is a float necessary? From the PR description:
Instead of counting the number of events near expiry, it now measures the "TTL budget"
A budget in integer seconds is precise enough for our needs
There was a problem hiding this comment.
func (d *DurableEmitter) metricsLoop() {
mc := d.cfg.Metrics
poll := mc.PollInterval
if poll <= 0 {
poll = 500 * time.Millisecond
}
ctx, cancel := d.stopCh.NewCtx()
defer cancel()
the default of pollInteval is every 10 seconds, which is the preferred cadence over every 1/2 second. IFIUC this is where we set the default, so the implementation doesn't match the comment. Let's set to 10 seconds please
This pull request refactors how the durable emitter tracks and reports queue expiry pressure metrics. Instead of counting the number of events near expiry, it now measures the "TTL budget"—the remaining time before the oldest pending event expires. This simplifies the code and provides a more direct signal for alerting on dead-letter queue (DLQ) pressure.
Metrics and Observability Improvements:
NearTTLCountmetric (number of events near expiry) with a newTTLBudgetmetric, which tracks the remaining TTL headroom for the oldest pending event. This is now reported asqueue.ttl_budget_seconds.DurableQueueStatsstruct and theObserveDurableQueueinterface to remove thenearExpiryLeadparameter and field, and instead use the newTTLBudgetfield.PgDurableEventStore,MemDurableEventStore, and metric wrappers) to remove all logic related to "near expiry" windows and focus only on TTL budget calculation.